CLAIMS 



What is claimed is: 

1 . A method for transmitting a mixed media data stream in packets, including 

audio and video objects, between a sender and a receiver through a connection over a 
network, the method comprising: 

monitoring, at the receiver, transmission characteristics of the cormection between the 
server and the receiver; 

estimating available bandwidth at the sender based upon the transmission characteristics 
of the connection monitored at the receiver; 

allocating a global buffer for the mixed media data stream to be transmitted from the 
sender to the receiver as a function of the estimated available bandwidth at the sender; 

pre-encoding a portion of each Video Object Plane (VOP) in the global buffer with 
respect to a quantization parameter (QP) of the VOP; 

encoding the VOP in the global buffer based on the QP; 

updating a rate distortion model based upon the QP and packet loss rate; 

performing a frame skipping function after the VOP encoding; and 

transmitting from the sender to the receiver the encoded video object plane in the global 
buffer at a regulated sender transmission rate from the sender as a fimction of the estimated 
available bandwidth at the sender. 
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2. The method as defined in Claim 1, further comprising: 
receiving the encoded video object plane at the receiver from the connection; 
demultiplexing the encoded video object plane into coded video and audio streams; 
inputting the coded video and audio streams, respectively, into video and audio 
decoders; 

inputting the decoded video and audio streams to a media mixer; and 
inputting the mixed video and audio streams output from the media mixer to an output 
device. 



' 3 . The method as defined in Claim 1 , wherein pre-encoding a portion of each VOP with 

Z respect to the QP of the VOP further comprises adjusting the QP of the VOP. 



4. The method as defined in Claim 3, wherein the QP of the VOP is adjusted with 
respect to a texture parameter, r, as the number of bits which will be used to encode the VOP 
wherein: 



p. X MAD p X 
r = + ^ 



MAD 



QP QP^ ' 

pi and p2 are control parameters; and 

MAD is a mean absolute distortion in which a total target bit rate for all objects in 
the global buffer are allocated proportionally to motion, size, and square of MAD. 



5. The method as defined in Claim 4, wherein the adjusting of the QP of the 
VOP is performed by changing the QP to a values in a range from 1 to 3 1 depending upon 
the estimated available bandwidth at the receiver. 
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6. The method as defined in Claim 1, wherein updating a rate distortion model 
based upon the QP and packet loss rate comprises: 

predicting the number of bits, ri, to encode the ith VOP, and is given by: 

^ I I L^_2-l L* 

'■" ef. QP^ ' 

the distortion, d, is estimated hy d, = (q.).xQR + (q ).x QP^ + (^^ ) • x ^- x (^r ) • . 
wherein: 

qi, q2 and qs are control parameters; and 

the packet loss rate (Pl )i is an estimate of that the probability that the 
ith transmission of data from the sender will be lost; and 
minimizing the overall distortion, D, for each encoded VOP by Z) = X^- . subject to 

R = Y.r,<Rj,, where Rt is the total bit budget for the current time instant. 
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7. The method as defined in Claim 1, wherein: 

the sender sends data to the receiver in through a connection over a packet switched 
network in a sender packet having a sender header that includes: 
a packet sequence number; 

a timestamp indicating the time when the sender packet was sent (STl); and 

the size of the sender packet (PacketSize); 
the receiver sends data to the sender through the connection over the packet switched 
network in a receiver packet having a receiver header that includes: 

the thne interval that the sender packet spent in the receiver side ( A RT); 

the timestamp of the sender packet sent from the sender (STl); 

an estimate, calculated by the receiver, of a packet-loss rate; and 

the rate at which data is received at the receiver; 
monitoring transmission characteristics of the connection between server and receiver 
comprises: 

estimating a round trip time of the sender packet from the sender to the 
receiver (RTT) based on STl and ^ RT; 

estimating a time out interval (TO) before which the sender should 
retransmit to the receiver a sender packet of data that has not been received 
by the receiver; 

estimating a probability that a packet of data will be lost (Pl); 

estimating the present available network bandwidth at which the receiver 
can receive data from the sender (rvcrate) as a function of the PacketSize, the 
RTT, the Pl, and the TO; 
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deriving the present sending rate of data from the sender to the receiver 



(currate); 

setting an updated sending rate of data from the sender to the receiver 
(currate)^ wherein: 



if rcvrate is greater than currate , then deriving currate as a 



function currate , PacketSize, and RTT; and 



if rcvrate is not greater than currate , then setting currate to be 



less than rcvrate. 



8. The method as defined in Claim 7, wherein: 



RTT = axRTT + {\~a)x{now-ST\-I^T);^nd 

now is the timestamp indicating the time at which the receiver packet was received in 
the sender; and a is a weighting parameter. 



9. The method as defined in Claim 7, wherein: 
TO = RTT + (kxRTTVAR); 
k is a constant; 



RTTVAR = ^2 X RTTVAR + (1 - ^2 ) x RTT - (now - STl ~ ART) 



RTTVAR is the current variation in the round trip time of the sender packet from the 
sender to the receiver (RTT); 

a2 is a weighting parameter; and 



RTTVAR is a smoothed estimate of RTTVAR . 
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The method as defined in Claim 7, wherein Pl is derived by a Gilbert 
The method as defined in Claim 10, wherein: 

A 

9 . 

A A ' 

p+q 

Xi takes 1 if the ith sender packet has arrived successfully at the receiver; 
Xi takes 0 if the ith sender packet is lost; 

q = P[X,=0\X,_,=l]; 

A 

p is an estimate of p; 

A 

^ is an estimate of q; and 

A A 

noi is the number of times in an observed time series when one 
follows zero; 

nio is the number of times when zero follows one; 
no is the number of zeros; and 
ni is the number of ones. 
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12. The method as defined in Claim 11, wherein: 

the Pl is further smoothed by a filter that weights the n most recent measured 
packet loss rates by: 

7=0 

P 

^'^"•^ is the measured packet loss rate in the (i-j)th time interval; 
two set of weighting parameters are defined as follows: 





WO 


Wl 


W2 


W3 


W4 


W5 


W6 


W7 


WSl 


1.0 


1.0 


1.0 


1.0 


0.8 


0.6 


0.4 


0.2 






wo 


Wl 


W2 


W3 


W4 


W5 


W6 


W7 


WS2 


1.2 


1.2 


1.0 


1.0 


0.8 


0.5 


0.3 


0.1 



; and WS2 is used for Wj when the actual packet loss rate is less than 
half of the measured packet loss rate, otherwise WSl is used for Wj. 

13. A computer-readable media comprising computer-executable instructions for 
performing the method as recited in Claim 1 . 
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14. A method for transmitting a mixed media data stream in packets, including audio and 
multiple video objects (MVOs), between a sender and a receiver through a connection over a 
network, the method comprising: 

monitoring transmission characteristics of one or more encoded video object 
planes through the connection between the sender and the receiver; 

estimating, from the transmission characteristics, an available bandv^dth (Rt) at 
the sender; 

allocating, as a function of the Rt, a portion of the mixed media data stream to a 
global buffer; 

encoding a video object plane from the global buffer based upon a rate distortion 
function that accounts for packet loss rate between sender and receiver; 

updating the rate distortion function based upon results of the encoded video 
object plane and upon a memory containing results of one or more previously 
encoded video object planes; 

after the encoding the MVOs in the video object plane, performing a frame 
skipping function; and 

transmitting, at the estimated available bandwidth, the encoded video object 
plane from the sender to the receiver. 
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15. The method as defined in Claim 14, wherein allocating a portion of the mixed media 

data stream to a global buffer comprises: 

W = max(((^ +B )xR^/R IF), 0) , as the global buffer size 

cur prev prev 1 I old 11 

Roid/2, is changed to Rt/2, wherein: 

Bprev is the number of bits spent in the previous time instant Bprev, 

Roid/2 is the previous size of the global buffer; 

Wprev is the previous occupancy of the global buffer; and 

F is the video frame rate. 



16. The method as defined in Claim 14, wherein allocating a portion of the mixed media 
data stream to a global buffer comprises the allocation of an output target rate from the 
global buffer among each of video and audio data streams so as to yield the target bits for an 
individual object in the data stream. 



17. The method as defined in Claim 14, further comprising: 
receiving the encoded video object plane at the receiver from the connection; 
demultiplexing the encoded video object plane into coded video and audio streams; 
inputting the coded video and audio streams, respectively, into video and audio 
decoders; and 

inputting the decoded video and audio streams to a media mixer; and 
inputting the mixed video and audio streams output from the media mixer to an output 
device. 
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18. A computer-readable media comprising computer-executable instructions for 
performing the method as recited in Claim 14. 
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1 9. A method for transmitting a mixed media data stream including multiple video 
objects (MVOs) between a sender and a receiver through a connection over a packet 
switched network, the method comprising: 

transmitting from the sender a sender packet of data to the receiver, the sender packet 
including a sender header that includes: 
a packet sequence number; 

a timestamp indicating the time when the sender packet was sent (STl); and 
the size of the sender packet (PacketSize); 
transmitting from the receiver a receiver packet of data to the sender, the receiver packet 
including a receiver header that includes: 

the time interval that the sender packet spent in the receiver side ( A RT); 
the timestamp of the sender packet sent from the sender (STl); 
an estimate, calculated by the receiver, of a packet-loss rate; and 
the rate at which data is received at the receiver; 
estimating a round trip time of the sender packet from the sender to the receiver (RTT) 
based on STl and a RT; 

estimating a time out interval (TO) before which the sender should retransmit to the 
receiver a sender packet of data that has not been received by the receiver; 
estimating a probability that a packet of data will be lost (Pl); 
estimating the present available network bandwidth at which the receiver can receive 
data from the sender (rcvrate) as a ftinction of the PacketSize, the RTT, the Pl, and the TO; 

deriving the present sending rate of data from the sender to the receiver ( cur rate ); 
setting an updated sending rate of data from the sender to the receiver (currate), wherein: 
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if rcvrate is greater than currate , then deriving currate as a function 



currate , PacketSize, and RTT; and 



if rcvrate is not greater than currate , then setting currate to be less than 



rcvrate. 



20, The method as defined in Claim 19, wherein: 



RTT = axRTT^{\'-a)x{now-ST\-MCr)\ 

now is the timestamp indicating the time at which the receiver packet was received in 



the sender; and 



a is a weighting parameter. 



21 . The method as defined in Claim 19, wherein: 



TO = RTT -\^(kx RTTVAR) ; 



k is a constant; 



RTTVAR = a2X RTTVAR + (1 - ^2 ) x RTT - (now - STl - ART) 



RTTVAR is the current variation in the round trip time of the sender packet from the 
sender to the receiver (RTT); 

a2 is a weighting parameter; and 



RTTVAR is a smoothed estimate of RTTVAR 



22. The method as defined in Claim 19, wherein Pl is derived by a Gilbert 

Model. 
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23. The method as defined in Claim 22, wherein: 



A 



p - ^ • 

L ^ ^' 

p+q 



Xj takes 1 if the ith sender packet has arrived successfully at the receiver; 
Xi takes 0 if the ith sender packet is lost; 
p = P[X,=^\\X^_,=Qi]; 



q = P[X=Q\X,_,=\l 



A 

jt? is an estimate of p; 



A 

^ is an estimate of q; and 



A A 

p = n^^jn^ and q = ^^q/^^ ? wherein: 



noi is the number of times in an observed time series when one 



follows zero; 



nio is the number of times when zero follows one; 
no is the number of zeros; and 



ni is the number of ones. 



lee@hayes 



50 



MSI-5S9US.PATAPP 



[ f 



24. The method as defined in Claim 23, wherein: 

the Pl is further smoothed by a filter that weights the n most recent measured 



packet loss rates by 



n-l 



IT ■ 

; ^'^"-^ is the measured packet loss rate in 



the (i-j)th time interval; 



two sets of weighting parameters are defined as follows: 







wo 


Wl 


W2 


W3 


W4 


W5 


W6 


W7 




WSl 


1.0 


1.0 


1.0 


1.0 


0.8 


0.6 


0.4 


0.2 





WO 


Wl 


W2 


W3 


W4 


W5 


W6 


W7 


WS2 


1.2 


1.2 


1.0 


1.0 


0.8 


0.5 


0.3 


0.1 



; and WS2 is used for Wj when the actual packet loss rate is less than half of 
the measured packet loss rate, otherwise WSl is used for Wj, 



25. The method as defined in Claim 24, wherein 



rcvrate - 



PacketSize 



RTTx pP^ /3+3xTOxP^x ^SP^ /8 x (1 + 32P^^ ) 
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26. The method as defined in Claim 19, wherein: 

lastchange is the timestamp indicating the time at which last adjustment to 
currate occurred; 

now is the present moment; and 
setting currate further comprises: 

if rcvrate is greater than currate , then currate = currate + 
(PacketSize/RTT) X multi, where multi = {now - lastchange)/RTT, and multi 
is constrained from 1 to 2; and 

if rcvrate is not greater than currate , then currate ^ a constant X 
rcvrate + (1- the constant) X currate . 

27. The method as defined in Claim 19, further comprising: 

allocating, as a function of the rvcrate, a portion of the mixed media data stream to a 
global buffer; 

encoding a video object plane from the global buffer based upon a rate distortion 
function that accounts for packet loss rate between sender and receiver; 

updating the rate distortion function based upon results of the encoded video object 
plane and upon a memory containing results of one or more previously encoded video object 
planes; 

after the encoding the MVOs in the video object plane, performing a frame skipping 
function; and 

transmitting, at the estimated available bandwidth, the encoded video object plane 
from the sender. 
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28. The method as defined in Claim 27, wherein: 

Rt is the total bit budget for the cxarrent time instant obtained from the rvcrate; and 
allocating a portion of the mixed media data stream to a global buffer comprises: 

W = max(((W +5 )xR^/R ..-R^ IF), 0) , as the global buffer 
cur prev prev^ T I old 11 

size Roid/2 is changed to Rt/2, wherein: 

Wcur is the occupancy of the global buffer; 

Bprev is the number of bits spent in the global buffer in the previous time 
instant, 

Roid/2 is the previous size of the global buffer; 

Wprev is the previous occupancy of the global buffer; and 

F is the video frame rate. 

29. The method as defined in Claim 28, wherein allocating a portion of the mixed media 
data stream to a global buffer fiirther comprises the allocation of a output target rate from 
the global buffer among each of video and audio data streams so as to yield the target bits 
for an individual object in the data stream. 
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30. The method as defined in Claim 28, wherein updating the rate distortion function 
based upon results of the encoded video object plane and upon a memory containing results 
of one or more previously encoded video object planes comprises a minimization of 

D = Y.oc.xd. , subject to i? = Z < Tfj^ ; wherein: 
It j 

I 

Rj = Z currate^ , 
i 

the sending rate of the ith media stream is , 

the distortion of ith media as , and the quality-impact parameter of the data stream is 

:F 3 1 . The method as defined in Claim 27, further comprising: 

receiving the encoded video object plane at the receiver from the connection; 
^^J demultiplexing the encoded video object plane into coded video and auto streams; 

f:l inputting the coded video and audio streams, respectively, into video and audio 

decoders; 

inputting the decoded video and audio streams to a media mixer; and 
inputting the mixed video and audio streams output from the media mixer to an output 
device. 

32. A computer-readable media comprising computer-executable instructions for 
performing the method as recited in claim 19. 
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33. One or more computer-readable media, comprising stored thereon: 

a first set of elements to describe a server in communication through a coimection of 
a packet switched network to a client; 

a second set of elements to describe the transmission of a mixed media data stream, 
including audio and multiple video objects (MVOs), from the server to the client through a 
connection over a packet switched network; and 
a code segment that, when executed: 

monitors transmission characteristics of the connection between server and 
receiver; 

estimates available bandwidth at the sender based upon the monitored 
transmission characteristics of the connection; 

allocates a global buffer for the mixed media data stream to be transmitted stream 
from the sender to the receiver as a function of the estimated available bandwidth at 
the sender; 

pre-encodes a portion of each Video Object Plane (VOP) in the global buffer 
with respect to a quantization parameter (QP) of the VOP; 

encodes the VOP in the global buffer based on the QP; 

updates a rate distortion model based upon the QP and packet loss rate; 

performs a frame skipping function after the VOP encoding; and 

transmits from the sender to the receiver the encoded video object plane in the 
global buffer at a regulated sender transmission rate from the sender as a function of 
the estimated available bandwidth at the sender. 
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34. One or more computer-readable media, comprising stored thereon: 

a first set of elements to describe a server in communication through a connection of 
a packet switched network to a client; 

a second set of elements to describe the transmission of a mixed media data stream, 
including audio and multiple video objects (MVOs), from the server to the client through a 
connection over a packet switched network; and 
a code segment that, when executed: 

monitors transmission characteristics of one or more encoded video object planes 
through the connection between the sender and the receiver; 

estimates, from the transmission characteristics, an available bandwidth (Rt) at 
the sender; 

allocates, as a function of the Rt, a portion of the mixed media data stream to a 
global buffer; 

encodes a video object pane from the global buffer based upon a rate distortion 
function that accounts for packet loss rate between sender and receiver; 

updates the rate distortion function based upon results of the encoded video 
object plane and upon a memory containing results of one or more previously 
encoded video object panes; 

after the encoding the MVOs in the video object plane, performs a frame 
skipping function; and 

transmits, at the estimated available bandwidth, the encoded video object plane 
from the sender to the receiver. 
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